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ABSTRACT 



An observation instrument was developed for use in assessing 
the competence of preservice technology teachers in Taiwan. The observation 
instrument was developed by using the nominal group technique to gather input 
from 5 experts in technology education and 60 junior high school teachers 
with a background in technology education. To evaluate the reliability of the 
new observation instrument, the teaching of five technology teachers was 
videotaped and evaluated by five experienced technology teachers and a 
technology professor. Input regarding the new observation instrument's 
validity was also obtained through a questionnaire. The experts confirmed 
that the new observation instrument did indeed have expert validity. The 
instrument's reliability was determined by calculating the Kendall 
coefficient, which was determined to equal 0.917 and to confirm that the 
instrument is indeed reliable. The new observation instrument contains 39 
closed- and open-ended items (evaluation criteria) in the following areas: 
teaching activities (explain the content of living technology, develop 
teaching activities, use various teaching techniques) ; teaching presentation 
(use good oral presentation techniques) ; classroom management (create a good 
learning climate, maintain good student discipline and behavior) lab 
management; and teaching evaluation and feedback. (Contains observation 
instrument . ) (MN) 
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ABSTRACT 



A new living technology curriculum at junior high schools 
in Taiwan was implemented in 1997. There is no observation 
instruments for assessing the teaching performance of pre- 
service technology teachers in Taiwan. Thus, it is very 
important to develop an observation instrument for assessing 
the teaching performance of pre-service technology teachers. 

The purpose of this study is to develop the observation 

instrument for pre-service teachers of living technology at 
junior high schools. This study was begun with developing 
instrument and its contents by the nominal group. The draft of 
evaluation .instrument was made, tested, and then revised by 
researchers. Then, the draft was revised three times by panel 
discussion of experts and the living technology teachers 
thoroughly to establish the validity of the observation 
instrument . 

Five pre-service teachers were taped to establish the 
reliability of the observation instrument. Then, the 
teaching process of tape was evaluated by five senior 
teachers and a professor of living technology. After that, 
the questionnaire was implemented to investigate the 
response about the usage of the observation instrument. 
Finally, the conclusion and suggestions were provided based 
on the result above. 
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Observation instrument for assessing 
pre- service technology teachers 

Need 

In Taiwan, according to the Teacher Education Law, 
a new evaluation system for teaching practice of pre- 
service technology teachers will be conducted in the 
calendar year of 1997. At the same time, a new living 
technology curriculum at junior high schools in Taiwan 
was implemented. The technology teaching is one of the 
important teaching event in a teacher education 
program. However, there is no observation instruments 
for assessing the teaching performance of pre-service 
technology teachers in Taiwan. Thus, it is very 
important to develop an observation instrument for 
assessing the teaching performance of pre-service 
technology teachers. 

Purpose of the Study 

The first purpose of this study is to develop an the 
observation instrument which has its validity for technology 
teaching. The instrument is designed to assess the competence 
of the pre-service teacher in technology teaching. The second 
purpose of this study is to apply the instrument in evaluating 
technology teaching to establish its reliability. 
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Research Framework Method and Process 

method 

The methodology for the study included a literature 
review, nominal group technique which was described in table 
1 (Tsi-Li Kaung, 1995), panel discussion, instrument 
development, and instrument evaluation. Sixty-five (three times) 
experts were asked for evaluating technology teachers to set 
up the validity of the instrument. 

In order to set up the reliability of this instrument, 
the videos of technology teaching from five technology 
teachers were recorded. An evaluation was conducted by five 
technology teachers and a technology professor. At last, the 
questionnaire was sent out to investigate the response about 
the usage of the observation instrument. 

Table I . 



Method and Content of Nominal Group Technique 



Step 


Holder’s work 


Group work 


1 . hold meeting 


1. welcome participators 

2. describe goals & importance 
of the meeting 

3. define the roles of members 




2. produce idea 
by themselves 


1 . propose explored problems 
for 1 5 minutes 


1 . think about problems 
by oneself 

2. write down answers 


3. list idea by 
turns 


1 . present ideas on blackboards 
until all the ideas listed 


1 . present one idea 
each time 


4. clarify discussion 


' 1 . clarify all the idea to let 
each member understand 
its meaning 


1. propose unclear 
idea 
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5. list idea according 1 . collect evaluation card ] . list idea according to 

to priority for analysis priority or evaluation 

score 

Process 

The process of this research was indicated in Figure 1. 
The first phase of the literature analysis and related research 
including theory (Dei-Jang Chang, 1994), competencies (Data 
Working Group, 1995), and instruments (Cinquini, V. , Robutti, 
0., & Vincenzi, A. B., 1994; Estes, G.D., Stansbury, K. , & 

Long, C., 1990; Simpson, R. D., & Brown, D.R., 1977; South 

Carolina Education Improvement Task Force, 1981), was explored. 
The second phase of competencies, and criteria was establish 
according to literature review and nominal group discussion. 
The third phase of the instrument and manual was developed by 
the author. A video of technology teaching was recorded. The 
research team which included five members reviewed and 
evaluated the video. The research team met for twenty five 
hours to discuss and edit the instrument and manual. This team 
includes one principal who has doctor degree, two instructors 
who have master degrees, one teacher in school, and myself. All 
the five members of the team have had teaching experience. 
Their efforts aided with the development of the final version 
of the observation instrument and manual. 

Five experts at universities and sixty teachers in junior 
high shcools, with background in technology education, 
validated the instrument. This ascertained that the 
competencies and criteria of the instrument had expert validity. 
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Then, some technology teaching units were selected and the 
teaching of five technology teachers were recorded. Five 
experienced technology teachers and a professor evaluated 
videos of five technology teachers to set up the reliability of 
this instrument. As a result, the correlation coefficient was 
figured out. 

Finally, the investigation of questionnairing was 
implemented to get the response about the usage of the 
observation instrument. 



Review of instruments 



Nominal group discussion- 



set up criteria- 



develop instrument ► edit 

set up validity * set up reliability 



finish 



Figure 1 . Research Framework 

Content of the instrument and manual 

This observation instrument was divided into five parts. 
Each part had its percentage in whole score. They are: (1) 
teaching activities; (2) teaching presentation; (3) classroom 
management; (4) lab management; (5) teaching evaluation and 
feedback. It includes instrument, manual and its application. 
The instrument includes thirty-nine items of criteria. They are 
as follows: 
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observing instrument for assessing pre-service technology 
teachers 



Pre-service teacher: Evaluator: 

Date : Time : 

Description of Evaluation: 

This instrument is for assessing the performance of pre- 
service teachers in the technology education by observing the 
teaching . 

Criteria and Directions: 

1. The item of competence to be considered "pass" depends 
upon the pre-service technology teacher’ s performance of 
all the assessment items. 

2. A "very good" rating is equal to 5 scores. A "good" 
rating is equal to 4 scores. A "medium" rating is equal 
to 3 scores. "Bad" is equal to 2 scores. A "very bad" 
rating is equal to 1 score. If the student get 3 or more 
than 3 scores in each item of competence, his level of 
competence is considered as a "pass" score. 

3. If the total score of the competence items is 117 or 
more, then the pre-service technology teacher' s 
performance is considered a "pass." 

4 . The description of the criterion is presented in the 
"observation instrument manual for assessing the teaching 
performance of pre-service technology teachers. 

Evaluation result for this instrument: 

Scores: Pass □ , Fail □ 

Signature of evaluator: 

Phase A: teaching activities 

Competencies & Criteria very good, Good, Medium, Bad, very Bad 

5 4 3 2 1 

/ 

Competency A1 : to explain the content of living technology 

Criterion A1 . 1 : indicates the important teaching content of 
information and communication, construction 
and manufacture, and energy and transportation 
systems '(three technology systems) 

□ □ □ □ □ 

Criterion A1 . 2 : describes the application of three technology 
systems on our life 
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□ n non 

Criterion A1 . 3 : clarifies the learning difficulty and conceptual 
mistakes in three technology systems 

□ n □ n n 



Competency A2 : to develop teaching activities of living technology 

Criterion A2 . 1 : motivates students to learn technology when 

the class is begung □ □ □ □ □ 



Criterion A2 . 2 : makes use of examples to develop teaching activities 

□ □ n □ □ 

Criterion A2 . 3 : guides students to finish the manufacture of life- 
directed projects g n n I I I I 

Criterion A2 . 4 : selects appropriate resources for the project 

n □ □ □ □ 



Competency A3: to use various technology teaching techniques 



Criterion A3 . 1 : 
Criterion A3 . 2 : 



Criterion A3 . 3 : 
Criterion A3 . 4 : 



uses various teaching methods 
dexterously G G G G G 

divides students into small groups for implementing 
technology teaching activities according to their 
competencies G G I I I II I 

proper selection and use of teaching media 

□ □ □ □ □ 

effective use of the technology teaching 

time G G G G Q 



Phase A: Description: 



Phase A • Scores of Teaching activities: 



Phase B: teaching presentation 



Competency B1 : to present the teaching content using good oral 
presentation techniques 



Criterion Bl.l: uses simple and clear sentences or diagrams to 
present the teaching content □ □ □ □ □ 
Criterion B1.2: uses clear oral presentation of the 

teaching content □ □ □ □ □ 
Criterion B1 . 3 : presents the teaching content melodiously 



□ □ □ □ □ 

Criterion B1.4: lectures by means of appropriate volume 

□ □ □ □ □ 

Criterion B1.5: lectures by means of appropriate speed so students 
can follow G G G G G 



Competency B2 : to present the teaching content of living 
technology by means of body language 
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Criterion 
Criterion 
Criterion 
Phase B: 



B2 . 1 : uses appropriate sign language to forward 





message to students □ 


□ 


□ 


□ 


□ 


B2.2: 


forward message to students by 


means of 


appropriate 




facial expression □ 


□ 


□ 


□ 


□ 


B2.3: 


maintains appropriate 
poise □ 


□ 


□ 


□ 


□ 


Description : 











Phase B: Scores of Teaching presentation: 

Phase C: Classroom management 

Competency Cl: to construct good technology learning climate 



Criterion Cl . 1 : 
Criterion Cl . 2 : 



implements interactive technology teaching 

□ □ □ □ □ 

supervises students' project activities diligently 

□ □ □ □ □ 



Competency C2 : to maintain good students discipline in the classroom 



Criterion C2 . 1 : 
Criterion C2 . 2 : 
Criterion C2 . 3 : 



leads students to follow classroom rules 





□ 


□ 


□ 


□ 


□ 


stops students who 


violate 


classroom 


rules 


on a 


timely basis 


□ 


□ 


□ 


□ 


□ 


maintains student 


order in 


the 


classroom 






□ 


□ 


□ 


□ 


□ 



Competency C3 : to lead students to have good behavior 
performance in the classroom 



Criterion 


C3 . 1 : 


informs students of the 


expected 


behavior 








performance □ 


□ 


□ 


□ 


□ 


Criterion 


C3.2: 


compliments students on 


good behavior 










performance in class Q 


□ 


□ 


□ 


□ 



Phase C: Description: 



Phase C: Scores of Classroom management : 



Phase D: lab management 



Competency D1 : to maintain safety and sanitary conditions 
in the lab 



Criterion D1 . 1 : explains the rules of safety and sanitation 
the lab at the beginning of class 

□ □ □ □ 

Criterion D1.2: requests students to put on safe appliance 



in 



□ 




Criterion D1 . 3 : 
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□ □ □ □ □ 

guides students to practice safe procedure at all 

time CD □ [D CD Id 

Criterion D1 . 4 : maintains clear route for walking or working in the 
lab □ □ □ □ □ 



Competency D2 : to use and manage equipment and tools correctly 



Criterion D2 . 1 : operates various equipment and tools correctly 

□ □ □ □ □ 

Criterion D2 . 2 : guides students to make good use of personnel 
organization for the management of appliance, 
material, tools and equipment 

□ □ □ □ □ 

Criterion D2 . 3 : guides students to keep just used appliance, 

material, tools and equipment in order and clean 

them CD CD □ □ □ 

Phase D: Description: 



Phase D: Scores of lab management: 



Phase E: teaching evaluation and feedback. 



Competency El: to evaluate students objectively in technology 
teaching 

Criterion El . 1 : selects appropriate instruments according to 

teaching objectives CD CD CD CD CD 

Criterion El. 2: evaluates students' preliminary competencies 

when the class is begunCD CD CD CD CD 

Criterion El. 3: uses a variety of methodologies for evaluating 

the teaching performance of students according to 
teaching objectives CD CD CD CD CD 



Competency E2 : provides students feedback on evaluation results 



Criterion E2 . 1 : encourages students appropriately on 







evaluation result CD 


□ 


□ 


□ 


□ 


Criterion 


E2.2: 


guides students to self 
evaluation results 


-review 


based 


on 








□ 


□ 


□ 


□ 


□ 


Criterion 


E2.3: 


assign student lessons 


in accordance 


with 








evaluation results CD 


□ 


□ 


□ 


□ 



Other criteria (please describe): 
Phase E: Description: 



Phase E: Scores of Teaching evaluation & feedback: 



Evaluation result for this instrument: 



Scores : 
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Description: 



Observation Instrument manual for assessing the teaching 
performance of pre- service technology teachers 

Phase A: teaching activities 

Competency A2 : to develop teaching activities of living 

technology 

Criterion A2 . 2 : makes use of examples to develop 

teaching activities 

Description: The content of technology teaching 

activities in the classroom/lab includes 
information and communication, 
construction and manufacture, energy and 
transportation. Thus, teachers should 
make use of positive and negative 
examples to develop the teaching activity 
of the project which related to the above 
technological systems. 

Criterion A2 . 3 : guides students to finish the 

manufacture of the life-directed projects 

Description: Teachers should teach students how to 
manufacture the project. Therefore, the 
process of the manufacture need to be 
explained very clearly by teachers. 

Criterion A2 . 4 : selects appropriate resources for the 
project 

Description: Teachers should select and provide enough 
appliance, equipment, instructional 
materials, books, catalog of technology 
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product and etc. Teachers might also ask 
students to bring related resources such 
as waste materials, appliances and other 
learning materials. This is the way for 
students to improve their competencies of 
comprehension and application. 



Description : 



(Note: additional information regarding other criteria and 
description can be obtained from the author) 

Results 

Validity 

The instrument had expert validity. Five technoogy experts 
at universities and sixty technology teachers in junior high 
shcools, validated the instrument for three times. 

Reliability 

The Kendall coefficient of concordance was equal to 
0.917 for the whole phases of the instrument. This 
ascertained that the competencies and criteria of the 
instrument had reliability. 

Questionnairinq 

The response of investigation from questionnairing about 
the usage of the observation instrument was good. 



Application 
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Stage 1 : Diagnosis of the Disadvantages . 
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Step 1: Self-diagnosis . The pre-service technology teacher may use the instrument to self- 
evaluate his own teaching performance. 

Step 2: Supervisors’ or helping teachers’ evaluation . In addition to the supervisor and 
helping teacher evaluation, the principal and other teachers should be encouraged to evaluate the 
pre-service technology teacher. 

Step 3: Students’ assisted evaluation . The student should make an evaluation report 
assessing his pre-service technology teacher’s teaching performance. 

Step 4: Counseling meeting . A counseling meeting will be held to inform the pre-service 
technology teacher about the evaluaton results of supervisors, helping teachers and students. 
Suggestions are made to help the pre-service technology teacher to improve his teaching. 

Stage 2: Preparing professional development. 

In order to improve the teaching, the pre-service technology teacher should write a 
proposal on his own professional development under the supervision of his supervisor or 
helping teacher. 

Stage 3: Implementing professional development. According to the proposal, the pre-service 
technology teacher would watch videos on teaching, participate in a seminar, and simulate the 
points of other teachers’ teaching to improve his own performance. 

Stage 4: Review conference . A conference would be held for reviewing and discussing the 
performance of the pre-service technology teacher after professional development. If the pre- 
service technology teacher lacks performance in teaching, he should be advised on how to 
implement a professional development program that will assist him in teaching development. 

Conclusions and Recommendations 




Conclusions 
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This study drew the following four conclusions: 

l.It was determined that the following phases were required to 
assess the performance of the pre-service teacher in 
teaching of living technology at junior high schools by 
observing: (1) teaching activities; (2) teaching presentation; 

(3) classroom management; (4) lab management; (5) teaching 
evaluation and feedback. 

2 . The instrument includes the items of closed and opened 
questions in order to evaluate the whole performance of pre- 
service technology teachers. 

3 . The instrument is not only for formative evaluation but also 
summative evaluation. 

4 . The instrument of portfolio's evaluation is required for 
implementing the formative evaluation, just as buying or 
planning equipment, planning teaching before teaching in 
addition to the observation instrument of technology teaching. 

Recommendations 

This study recommended the following: 

1. Major objectives of the instrument: Preparing professional 

development should be the major objectives of the 
instrument, instead of only evaluating. 

2. Reference for evaluation: The instrument developed by this 

study might provide a reference for the evaluation of the 
pre-service teacher in living technology. 

3. Making good use of the instrument's manual: Before applying 
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the instrument, the manual of this instrument should be 
thoroughly reviewed and understood in order to make good use 
of the instrument. 

4. Further development of instruments: It is necessary for the 

further study to develop portfolio instrument for assessing 
the process of teaching practice for pre-service teachers in 
technology teaching. 
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